Private Preview Feedback: GitHub Enterprise Importer (GEI) can migrated repositories with GitHub owned blob storage

[boylejj]

GitHub Enterprise Importer (GEI) can migrate repositories with GitHub owned blob storage

Using GitHub-owned storage with GEI

Congratulations on being a part of the GitHub-owned storage for GEI beta!

The GitHub Enterprise Importer (also known as GEI), performs migrations from GitHub products, Bitbucket Server, and Bitbucket Data Center into your organizations accessible via GitHub.com. Migrations with GEI are performed by a helpful CLI utility, or as individual GraphQL requests.

In the past, to migrate from GHES, Bitbucket Server, and Bitbucket Data Center, an external storage provider was required to perform migrations. With the introduction of GitHub-owned storage for GEI, this is no longer a requirement. Migrations can now be performed with archives uploaded directly to GitHub.com!

Limitations

1. Migrations using GitHub-owned blob storage with GEI are gated by a feature flag during private preview. If you’re interested in participating in this private preview, please reach out to your GitHub account manager or contact our sales team to have this feature enabled for your enterprise.
2. This feature is not available for migration to GitHub Enterprise Cloud with data residency at this time.

Security practices and managing your archives with GitHub-owned storage

We designed every step of the GitHub-owned storage flow with security being paramount. Archives are uploaded to organizations, and the uploading user is authorized with the migrator role for the organization, just like the rest of GEI. The storage is explicitly write-only, and downloads from GitHub-owned storage are not possible. Only basic metadata is accessible to users with a migrator role for the organization, which includes the file name, size, upload date, a GUID attributed to the archive, and a special GEI URI that represents the archive.

After a migration completes, the archives that were used are immediately deleted. If an archive is uploaded, but is not used with a migration, the archive is deleted after 7 days. In addition, our GraphQL API has been extended to be able to query basic metadata for archives uploaded to an organization, and to be able to delete archives at-will. The GraphQL endpoints only allow querying and deletion of archives for users that have the migrator role for their organizations.

Using GitHub-owned storage with the GEI CLI

Using GitHub-owned storage with the GEI CLI is a snap! Make sure you are running version v1.9.0 or greater of the gei or bbs2gh extension, then simply pass the --use-github-storage option to your gh gei or gh bbs2gh commands. The CLI will then be performing migrations using GitHub-owned storage! There's nothing more to it!

Using GitHub-owned storage via individual API calls

If you're looking to cater to a more specific flow with your migrations, you may want to use the GEI GraphQL API. This section serves as an addendum to the public Migrating repositories from GitHub Enterprise Server to GitHub Enterprise Cloud documentation, and includes details on uploading and enqueuing migrations using GitHub-owned storage.

Compared to the public documentation, these are the steps for performing migrations while using GitHub-owned storage:

1. Create your personal access token
2. Get the ownerId for the destination organization
3. Set up blob storage
4. Set up a migration source in GitHub Enterprise Cloud
5. Generate migration archives on your GitHub Enterprise Server instance
6. Upload your migration archives to GitHub-owned storage (a different step)
7. Start your repository migration
8. Check the status of your migration
9. Validate your migration and check the error log

For step 6, there are two ways to upload your archives: single POST requests for archives up to 5 GiB, and via a multipart upload flow for archives between 5 MiB and 30 GiB. Depending on the migration, it may use one archive, or separate archives for Git data and repository metadata (i.e. pull requests, issues, etc.). In any case, perform the uploads as necessary, and keep track of the GEI URIs from your uploads to continue with a migration.

In step 7, when starting your migration, simply use the GEI URIs in the gitArchiveUrl and metadataArchiveUrl fields, and migrations will be performed using the archives uploaded to GitHub-owned storage!

And remember, archives are deleted after every migration that uses them, even for failed migrations, so don't forget to perform new archive uploads for every migration!

Performing single uploads to GitHub-owned storage (step 6, <5 GiB)

To perform a single upload, simply submit a POST request with your archive as the POST data to:

• https://uploads.github.com/organizations/{organization_id}/gei/archive?name={archive_name}

Substitute {organization_id} with your organization database ID. The ID can be found in the "id" key from https://api.github.com/orgs/your-organization-login. Please note that the database ID should be an integer (i.e. 9919), and not a GraphQL node ID (i.e. MDEyOk9yZ2FuaXphdGlvbjk5MTk=).

Also substitute {archive_name} with a filename of your choice that represents your archive. This value sets the "name" field on the archive metadata for easier management as a user, and the value makes no functional difference in your migrations.

Make sure to also include your PAT and a content type in the request headers, too. The content type is required, although it can be always set to application/octet-stream. Here's an example of the request headers necessary for the POST request:

Authorization: Bearer ghp_12345
Content-Type: application/octet-stream

The API will accept your data as raw binary data, so be sure to not use multipart forms. Multipart forms append headers and footers in the request, which will be accepted literally as part of the archive. Also keep in mind that multipart forms are also different than the multipart upload flow that is also detailed in this guide. For more information on multipart forms and how they append unwanted data to your archive, see the MDN documentation on multipart forms. Inspect the "size" value in the JSON response: it should match the file size on disk.

Note: If you are using Insomnia, please be aware that imported cURL requests might have been configured as a multipart form, which will not upload archives correctly. When creating this request, make sure to select "HTTP Request", set the request type to "POST", and in the "Body" tab, ensure that the "File" type is selected. Ensure that the headers above have been added, too.

Below is an example request using cURL to upload an archive. Ensure that your PAT is set in $GITHUB_PAT, replace {organization_id} with the target organization's database ID, and replace archive.tar.gz with the filename of the archive to upload. Make sure to keep the @ in the --data-binary value so the archive's contents are uploaded, and not a literal file name.

curl \
  --header "Authorization: Bearer $GITHUB_PAT" \
  --data-binary '@archive.tar.gz' \
  'https://uploads.github.com/organizations/{organization_id}/gei/archive?name=archive.tar.gz'

The response body will include a JSON object, like so:

{
  "guid": "363b2659-b8a3-4878-bfff-eed4bcb54d35",
  "node_id": "MA_kgDaACQzNjNiMjY1OS1iOGEzLTQ4NzgtYmZmZi1lZWQ0YmNiNTRkMzU",
  "name": "git-archive.tar.gz",
  "size": 33287,
  "uri": "gei://archive/363b2659-b8a3-4878-bfff-eed4bcb54d35",
  "created_at": "2024-11-13T12:35:45.761-08:00"
}

The "uri" value contains the GEI URI! This URI represents the uploaded archive, and will be used to enqueue migrations in step 7!

Performing multipart uploads to GitHub-owned storage (step 6, 5-30 GiB)

Multipart uploads are a little more involved, and follow a high-level flow like so:

1. Create the multipart upload with a POST request.
2. Upload 100 MiB of the archive as a PATCH request to a file part.
3. Repeat step 2 with additional parts until the file is fully uploaded.
4. Submit a PUT request to complete the migration.

Here are the steps in more depth! For all requests, ensure to include credentials in your headers with your PAT, i.e.

Authorization: Bearer ghp_12345

1. Start a multipart upload. Submit a POST request to https://uploads.github.com/organizations/{organization_id}/gei/archive/blobs/uploads, substituting {organization_id} with your organization ID. Include a JSON body like below with the archive name and size. The content type can remain as "application/octet-stream" for all uploads.

   {
     "content_type": "application/octet-stream",
     "name": "git-archive.tar.gz",
     "size": 262144000
   }

   This will return a 202 with an empty response body. In the response headers, the Location will look like this:

   /organizations/{organization_id}/gei/archive/blobs/uploads?part_number=1&guid=<guid>&upload_id=<upload_id>
   

   Keep this path handy, as it'll be used to upload your first file part. We'll call it the "next path." Also remember the GUID value, as it'll be used to enqueue a migration with the uploaded archive later.

2. Upload a file part. Upload 100 MiB of your file with a PATCH request to https://uploads.github.com/{location}, substituting {location} with the "next path" value. Like single uploads, ensure that the raw data is uploaded in the request body without using a multipart form. This will return a 202 with an empty response body. In the response headers, the Location will look like this:

   /organizations/{organization_id}/gei/archive/blobs/uploads?part_number=2&guid=<guid>&upload_id=<upload_id>
   

   Notice that this Location value is identical to the initial Location on step 1, except the part_number value is incremented. If this is the last file part necessary to upload the entire file, we'll need to make a request to the previous Location path (not the new one), so keep the old path as "last path," and replace "next path" with our new Location path.

3. Repeat step 2 until the upload is complete. Ensure that you are reading up to 100 MiB of the file at a time, and submitting requests to the new Location values with the incremented part_number values.

4. Complete the multipart upload. Submit a PUT request to the "last path" value from step 2 with an empty body. If all is well, you'll receive a 201, and your upload to GitHub-owned storage is complete! Your GEI URI can be constructed with the GUID from step 1 like this: gei://archive/{guid}.

Here is an example of a Ruby script that will perform the above flow using Faraday and Addressable:

#!/usr/bin/env ruby

require "faraday"
require "addressable"

class MultipartUploader
  UPLOAD_URL_TEMPLATE = Addressable::Template.new("https://uploads.github.com/organizations/{organization_id}/gei/archive/blobs/uploads").freeze
  PART_SIZE = 104857600  # 100 mebibytes.

  attr_accessor :archive_path, :organization_id, :guid, :next_path, :last_path

  def initialize(archive_path:, organization_id:)
    @archive_path = archive_path
    @organization_id = organization_id
  end

  def upload
    file = File.open(archive_path)

    puts "Starting multipart upload..."
    upload_start

    (1..).each do |part_number|
      body = file.read(PART_SIZE)

      break unless body

      puts "Uploading #{body.bytesize} bytes as part #{part_number}..."
      upload_part(body)
    end

    puts "Completing multipart upload..."
    upload_complete

    puts "Archive uploaded with GEI URI gei://archive/#{guid}"
  end

  private

  def upload_url
    @upload_url ||= UPLOAD_URL_TEMPLATE.expand(organization_id: organization_id)
  end

  def faraday
    @faraday ||= Faraday.new(url: upload_url) do |faraday|
      faraday.request :authorization, "Bearer", ENV["GITHUB_PAT"]

      faraday.response :json
      faraday.response :raise_error
    end
  end

  def upload_start
    body = {
      content_type: "application/octet-stream",
      name: File.basename(archive_path),
      size: File.size(archive_path)
    }

    response = faraday.post(nil, body.to_json)

    @next_path = response.headers["location"]
    @guid = Addressable::URI.parse(next_path).query_values["guid"]
  end

  def upload_part(body)
    response = faraday.patch(next_path, body)

    @last_path = next_path
    @next_path = response.headers["location"]
  end

  def upload_complete
    faraday.put(last_path)
  end
end

multipart_uploader = MultipartUploader.new(
  archive_path: "/example/archive/path.tar.gz",
  organization_id: 1234
)

multipart_uploader.upload

Querying and deleting archives from GitHub-owned storage via GraphQL

There are three ways to interact with your archives from GitHub-owned storage:

• Query all archives from an Organization object via the migrationArchives connection.
• Query a single archive with the MigrationArchive object.
• Delete a single archive with the deleteMigrationArchive mutation.

In all three requests, ensure that you include the GraphQL-Features header below during the beta period:

GraphQL-Features: octoshift_github_owned_storage

Querying all archives

Here's an example query that obtains the first 10 migrations for an organization by the organization node ID, which can be found in https://api.github.com/orgs/your-organization-login:

query(
  $id: ID!
) {
  node(id: $id) {
    ... on Organization {
      migrationArchives(
        first: 10,
        orderBy: { field: CREATED_AT, direction: DESC }
      ) {
        pageInfo {
          endCursor
          hasNextPage
        }
        nodes {
          id
          guid
          name
          size
          uri
          createdAt
        }
      }
    }
  }
}

Query variable	Description
id	The Organization node ID.

This will return a value like this:

{
  "data": {
    "node": {
      "migrationArchives": {
        "pageInfo": {
          "endCursor": "MjQ",
          "hasNextPage": false
        },
        "nodes": [
          {
            "id": "MA_kgDaACQzNjNiMjY1OS1iOGEzLTQ4NzgtYmZmZi1lZWQ0YmNiNTRkMzU",
            "guid": "363b2659-b8a3-4878-bfff-eed4bcb54d35",
            "name": "git-archive.tar.gz",
            "size": 33287,
            "uri": "gei://archive/363b2659-b8a3-4878-bfff-eed4bcb54d35",
            "createdAt": "2024-11-13T12:35:45Z"
          }
        ]
      }
    }
  }
}

If you prefer, you can also query your archives alongside your other organization fields. You can also look up your organization by the organization name, too. Here's an example query:

query(
  $login: String!
) {
  organization (login: $login) {
    login
    id
    name
    databaseId
    migrationArchives(
      first: 10,
      orderBy: { field: CREATED_AT, direction: DESC }
    ) {
      pageInfo {
        endCursor
        hasNextPage
      }
      nodes {
        id
        guid
        name
        size
        uri
        createdAt
      }
    }
  }
}

Query variable	Description
login	The organization name.

This will return a response like the one below:

{
  "data": {
    "organization": {
      "login": "github",
      "id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=",
      "name": "GitHub",
      "databaseId": 9919,
      "migrationArchives": {
        "pageInfo": {
          "endCursor": "MjE",
          "hasNextPage": false
        },
        "nodes": [
          {
            "id": "MA_kgDaACQzNjNiMjY1OS1iOGEzLTQ4NzgtYmZmZi1lZWQ0YmNiNTRkMzU",
            "guid": "363b2659-b8a3-4878-bfff-eed4bcb54d35",
            "name": "git-archive.tar.gz",
            "size": 33287,
            "uri": "gei://archive/363b2659-b8a3-4878-bfff-eed4bcb54d35",
            "createdAt": "2024-11-13T12:35:45Z"
          }
        ]
      }
    }
  }
}

Querying a single archive

To query an archive by the archive's node ID, use a query like the one below:

query(
  $id: ID!
) {
  node(id: $id) {
    ... on MigrationArchive {
      id
      guid
      name
      size
      uri
      createdAt
    }
  }
}

Query variable	Description
id	The MigrationArchive node ID.

This will return a response like below:

{
  "data": {
    "node": {
      "id": "MA_kgDaACQzNjNiMjY1OS1iOGEzLTQ4NzgtYmZmZi1lZWQ0YmNiNTRkMzU",
      "guid": "363b2659-b8a3-4878-bfff-eed4bcb54d35",
      "name": "git-archive.tar.gz",
      "size": 33287,
      "uri": "gei://archive/363b2659-b8a3-4878-bfff-eed4bcb54d35",
      "createdAt": "2024-11-13T12:35:45Z"
    }
  }
}

Deleting an archive

Archives are automatically deleted after they are used in a migration, and unused archives are always deleted after 7 days. This mutation can be useful if an archive was uploaded that is not planned to be used for a migration, and immediate deletion is desired.

To delete an archive, use the deleteMigrationArchive mutation like so:

mutation deleteMigrationArchive(
  $migrationArchiveId: ID!
) {
  deleteMigrationArchive(
    input: {
      migrationArchiveId: $migrationArchiveId
    }
  ) {
    migrationArchive {
      id
      guid
      name
      size
      uri
      createdAt
    }
  }
}

Query variable	Description
migrationArchiveId	The MigrationArchive node ID.

This mutation will return a response like the one below:

{
  "data": {
    "deleteMigrationArchive": {
      "migrationArchive": {
        "id": "MA_kgDaACQzNjNiMjY1OS1iOGEzLTQ4NzgtYmZmZi1lZWQ0YmNiNTRkMzU",
        "guid": "363b2659-b8a3-4878-bfff-eed4bcb54d35",
        "name": "git-archive.tar.gz",
        "size": 33287,
        "uri": "gei://archive/363b2659-b8a3-4878-bfff-eed4bcb54d35",
        "createdAt": "2024-11-13T12:35:45Z"
      }
    }
  }
}

[jmdoherty]

I attempted to do a single file upload and got this response:
{'message': 'Not Found', 'request_id': 'C202:31B3AE:3CB483:43EFCA:679BAAE8', 'documentation_url': 'https://docs.github.com/rest'}

The url used was https://uploads.github.com/organizations/{OurOrgDbId}/gei/archive?name=james-test-01. It's an EMU org, if that matters.

[AnttiHal]

Same for me.

[boylejj]

@jmdoherty, this private preview is gated by a feature flag. It appears you use failed because your account and org haven’t been granted access yet.

[danimoretti]

@boylejj How to request access to this tool?

[boylejj]

@danimoretti

Migrations using GitHub-owned blob storage with GEI are gated by a feature flag during private preview. If you’re interested in participating in this private preview, please reach out to your GitHub account manager or contact our sales team to have this feature enabled for your enterprise.

[robandpdx]

I have built a github cli extension that does all of these operations...
https://github.com/robandpdx/gh-blob